While software systems continue to grow in size and complexity, business demands continue to require shorter development cycles. This has led some software developers to compromise on functionality, time to market, and quality of software products. Furthermore, the increased schedule pressures and limited availability of resources and skilled labor can lead to problems such as incomplete design of software products, inefficient testing, poor quality, high development and maintenance costs, and the like. This may lead to poor customer satisfaction and a loss of market share for companies developing software and other products.
To improve product quality, many organizations devote an increasing share of their resources to testing and identifying problem areas related to software and the process of software development. Accordingly, it is not unusual to include a quality assurance team in software development projects to identify defects in the software product during and after development of a software product. By identifying and resolving defects before marketing the product to customers, software developers can assure customers of the reliability of their products, and reduce the occurrence of post-sale software fixes such as patches and upgrades which may frustrate their customers.
Testing and identifying problem areas related to software development may occur at different points or stages in a software development lifecycle. For example, a general software development lifecycle includes a high level requirements/design review, a detailed requirements/design review, code inspection, unit test, system test, system integration test, potentially a performance test, and typically, a user acceptance test. Moreover, as the software development lifecycle proceeds from high level requirements/design review to user acceptance test, costs for detecting and remedying software defects generally increases, e.g., exponentially.
In an effort to reign in cost and time overruns, organizations sometimes develop one or more test plans that consolidate ideas on how to allocate effort for a test project. Test plans may include efforts regarding how to design testing activities and focus for the test project. Planning for a test project (e.g., testing project planning, test project planning, project test planning) normally contains two key levels of planning: macro planning and micro planning.
Macro planning is typically achieved using a top-down approach. Macro planning is most effectively performed in the early stages of a project, and is usually accomplished by comparing the current project to the most appropriate available historical project. The result of macro planning is a high level plan for how to allocate effort and how to design the testing activities and focus. The challenges to macro planning frequently include the ability to find a sufficiently similar historical project on which to base planning decisions for the current test project.
Micro planning is typically achieved using a bottom-up approach, and generally includes very detailed plans for every test to be performed after the Unit Test. For example, a micro plan may define how to run a particular test, including the date(s) for the test, the personnel involved with the test, what to do with the results of the test, etc. As any project moves forward in time, more detailed information (e.g., requirements) become available. Very mature organizations might be able to effectively leverage micro planning by looking at the different characteristics of requirements (e.g., risk, size, complexity associated with each). When an organization can do this, they are able to produce a more granular and precise estimation of the effort required, as well as the specific optimum test focus of each defined activity in the macro plan. However, given the pace of schedules, it is increasingly challenging for projects to produce accurate, timely, and cost effective micro plans.
Current industry practice is to create a macro plan and a micro plan in isolation from one another, if one or both types of plans are even created at all. Because macro plans and micro plans are created separately from one another, common practices involving macro top-down and micro bottom-up test planning produce plans having disjointed perspectives. As a result, macro plans and micro plans often do not synchronize with one another due to several factors. One factor may be that different testing levels or activities are often defined differently in the macro and micro plans. For example, micro plan test activities do not necessarily map to those defined in the macro plan due to overlapping schedules required by schedule pressure, shared test environments required by limited infrastructure, or other constraints that frequently are only identified after macro planning is complete. Early in the life cycle when the macro plan is being developed, very little of the detailed information necessary to develop a micro test plan exists, e.g., cost, effort, schedule, and quality targets, etc. This can lead to stark differences between the macro and micro plans.
Another factor that contributes to divergence between macro and micro plans is that different resources (e.g., people) normally perform the macro and micro planning functions. For example, macro planning is often performed by a project manager or consultant, with input from a test strategist or architect. On the other hand, micro planning is typically performed after the macro planning and by different resources, such as the assigned test lead and/or test team personnel. Frequently there is little or no attention paid to ensuring the micro and macro plans are in synchronization throughout the duration of the project.
Another factor that contributes to the isolated and separate nature of macro and micro plans is that different tools are often used for macro and micro planning. For example, macro test planning typically involves some kind of scheduler software such as Microsoft Project, whereas effective micro test planning requires sophisticated requirements management tooling and reporting capabilities. No existing macro or micro tools are designed to integrate with one another, and no integrated tool currently exists.
Complex system development is very expensive and high risk. Due in part to the disjointed nature of test planning described above, a majority of defects are often found later in the life cycle where the cost to fix such defects increases exponentially with time. Test projects are planned inefficiently and/or ineffectively because there is no solution that provides the real time insight necessary to find and fix defects as early as possible.
Although scheduling software can help allocate resources on simple projects, the task of optimally staffing test execution projects is a more complex problem to solve due to the unknown impact of blocking defects resulting from test dependencies. If test resource allocation is not carefully constructed and maintained, a test project can very quickly find themselves in a situation where multiple resources may be delayed or blocked entirely from making any progress for unacceptably long periods of time. In these cases, test costs relative to benefits received are significantly higher than they should be, and the negative impact to cost and schedule is typically severe.
Conventional test planning tools and methods do not provide a mechanism to model alternative test scenario planning for the purposes of comparing them and determining the optimal balance of cost, risk, quality and schedule. As a result, “what if” alternative test planning typically is not performed by most projects since it is largely a manual task and too labor intensive to be delivered in real time for projects to benefit from the information.
Moreover, there is no model in the industry that is capable of predicting the number, severity, and cost of defects. Yet, increasingly, project stakeholders could make better decisions if this information could be made available to them in a timely way.
Furthermore, the differences between historic project information and the current project frequently result in inaccuracies between projections based on historical data compared against the actual current project results. Even further, the approach of using historical projects for estimation provides no guidance or useful insight into how to best adjust plans while the project is underway to reflect changed conditions. As a result, detailed estimation planning and/or ongoing calibration is rarely actually performed on many of the large and complex efforts that would most benefit from it.
Additionally, there are no industry wide models available to provide appropriate expected distributions of defects uncovered in System Integration Testing (SIT). As a result, SIT testing tends to be one of the most expensive kinds of testing relative to the benefit received. At the same time, SIT often is the most important testing phase to ensure a successful move to production for complex system integration projects.
As a result of the above-noted difficulties in test planning, macro plans and micro plans, if created at all, are often set aside and ignored soon after their creation. Projects often begin with the intent of developing and following the plans. However, as problems arise and real actions inevitably deviate from the plans, an escalation can occur where one deviation from the plans leads to another deviation which leads to another, and so forth. Soon, the plans are discarded and the project deals with problems ‘on the fly’ as they occur (i.e., without any organized plan). This, in turn, often leads to cost and time overruns, which ultimately frustrates the customer (e.g., end user).
Accordingly, there exists a need in the art to overcome the deficiencies and limitations described herein above.